Papers with quantitative pruning
Linguistically Grounded Analysis of Language Models using Shapley Head Values (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for probing language models for morphosyntactic constructions are not well understood . language models gain knowledge of grammatical phenomena during pretraining, but exactly how this knowledge is encoded is not well established. |
| Approach: | They propose a method for probing language models via Shapley Head Values . they use a BLiMP dataset to test their method on linguistic constructions based on a Shaply Head Value method . |
| Outcome: | The proposed method can be used to investigate linguistic knowledge in language models . it shows that attention heads responsible for processing related linguistic phenomena cluster together . |